Compromise Strategies for Action Selection: Supplementary Material

The dynamic programming technique used in the paper overlays a grid of points on top of the problem space and calculates the maximum expected utility of each location given optimal future actions. This is done recursively starting at the target locations and moving outward until values for all points are calculated, and then repeating the process until the values converge. Using the equations given in the main paper, the expected utility of each grid point in the environment can be calculated provided the EU(Aθ|ta, tb, λ′) function can be accurately determined and θ can be found. As the EUs are only known at grid points, the local EU function must be interpolated from the EU values of the surrounding points in order to determine the EU of a movement in an arbitrary direction. Using the interpolated surfaces for the local values of EU(Aθ|ta, tb, λ′), the value of θ can be determined by searching for the angle that maximises the function described by equation (3). Once the expected utility is determined for a grid point, its value is then used to calculate the expected utility of its neighbouring grid-points. This process is repeated until values are determined for all the grid points. Because the estimated utility value can change for a point when the values of its neighbours change, the values of all the points are repeatedly re-estimated until the values stabilise. The pseudocode for the algorithm is given in figure 4. Implementation details follow.